Adaptive Neural Network Representations for Parallel and Scalable Bayesian Optimization

Bayesian optimization has emerged as a powerful, new technique for interpolating and optimizing a wide range of functions which are expensive to compute. The primary tool of Bayesian optimization is the Gaussian process, which permits one to define a prior belief, which is then transformed into a posterior through sequential sampling of points. Unfortunately, Gaussian process interpolation suffers from a major computational bottleneck that makes it prohibitively expensive to use in large-scale optimization routines. In this work, we investigate deep neural networks for learning a representation of the space for which a much less computationally expensive interpolation algorithm may be used. Our implementation adaptively updates the representation of the neural network as more data becomes available over time. Since this neural network-based approach is parallelizable, we are able to exploit simultaneous function evaluation and network parameter learning. We show that this neural network-based approach leads to many of the same appealing properties of Gaussian processes, and which scales only linearly in the number of observations ceteris paribus. We additionally seek to provide a theoretical explanation regarding why an optimization algorithm of this form functions at all. For this, we develop a probabilistic theory of martingales and apply it to a performance metric for the neural network-based approach to Bayesian optimization. We find that the performance of the deep neural network is consistent with the predictions of our probabilistic theory. We therefore speculate on the kind of representation that is learned by the network in its final hidden layer. We additionally show that our approach is easily extended to a distributed environment, which results in faster optimization. Numerical experiments demonstrate the efficacy of using deep networks for learning representations amenable to computationally inexpensive interpolation methods.